Optimization of the Error Entropy Minimization Algorithm for Neural Network Classification

One way of using entropy criteria in learning systems is to minimize the entropy of the error between the output of the learning system and the desired targets. In our last work, we introduced the Error Entropy Minimization (EEM) algorithm for neural network classification. There are some sensible aspects in the optimization of the EEM algorithm: the size of the Parzen Window (smoothing parameter) and the value of the learning rate parameter are the two most important. In this paper, we show how to optimize the EEM algorithm by using a variable learning rate during the training phase. INTRODUCTION The Error Entropy Minimization (EEM) algorithm minimizes the Reniy’s Quadratic Entropy of the error between the output of the neural network and the desired targets. Reniy’s Quadratic Entropy is used because, when using a nonparametric estimate of the probability density function (pdf) by the Parzen Window method, a simplified expression for the entropy is obtained. The size of the Kernel window, h, used in Parzen Window method is one of the two most sensible aspects in the implementation of the EEM algorithm. The other important aspect is the backpropagation learning rate. We implemented a variable learning rate, similar to the one used in the MSE algorithm, and the results were better than the ones obtained with the previous algorithm. RENYI'S QUADRATIC ENTROPY AND BACK-PROPAGATION ALGORITHM Renyi extended the concept of entropy introduced by Shanon and defined the Renyi's α entropy, applied to continuos random variables, as (Renyi 1976): [ ] ∫ − = C R dx x f H α α ) ( log (1) Renyi's Quadratic Entropy, 2 = α , in conjunction with the Parzen Window pdf estimation with gaussian kernel allows, the determination of entropy in a non-parametric, practical and computationally efficient way. The Parzen window method estimates the pdf ) (x f as ∑ = ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − = N i i m h x x K Nh x f 1 1 ) ( (2) where N is the number of data points, K is a kernel function, m is the dimensionality of vectors x ( m x R ∈ ) and h the bandwidth or smoothing parameter. We usualy use the simplest Gaussian kernel with zero mean and Σ equal to I (the identity matrix) ⎟ ⎠ ⎞ ⎜ ⎝ ⎛− = x x x G T m 2 1 exp